Providing suggestions within a document

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing suggestions within a document. In one aspect, a method includes obtaining textual input provided to a document editing application by a user device, the textual input being provided to the document editing application for inclusion in a document; identifying performance measures associated with the current editing session for the document, each performance measure being based on session data obtained from the user device during a document editing session, the session data being for the textual input and prior text that was included in the document prior to the textual input; providing the performance measures as input to a suggestion model that was trained using historical performance measures identified in performance logs for historical document editing sessions of users; and throttling textual suggestions during the current editing session based on the output of the suggestion model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S.application Ser. No. 14/275,333, filed on May 12, 2014, the entirecontents of which are hereby incorporated by reference.

BACKGROUND

This specification relates to providing suggestions within a document.

Document editing applications provide authors with many tools to assistusers with drafting documents, such as word processing documents, e-mailmessages, and network blog posts. The assistance provided by these toolsvaries greatly, from design assistance tools for designing layouts andformatting text, to revision tracking tools for tracking documentchanges. Other tools provide assistance based on the text included inthe document, such as spell checking tools that check text for spellingerrors, and grammar check tools that check text for grammatical errors.Each tool provided by a document editing application is generallydesigned to enhance the user's experience in drafting a document.

SUMMARY

This specification describes technologies relating to providingsuggestions for inclusion in a document.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof obtaining, during a current editing session, textual input providedto a document editing application by a user device, the textual inputbeing provided to the document editing application for inclusion in adocument; identifying, during the current editing session, one or moreperformance measures associated with the current editing session for thedocument, each performance measure being based on session data obtainedfrom the user device during a document editing session, the session databeing for the textual input and prior text that was included in thedocument prior to the textual input; providing the one or moreperformance measures as input to a suggestion model that was trainedusing historical performance measures identified in performance logs fora plurality of historical document editing sessions of a plurality ofusers; and throttling textual suggestions during the current editingsession based on the output of the suggestion model, each textualsuggestion comprising one or more words to be suggested for inclusion inthe document. Other embodiments of this aspect include correspondingsystems, apparatus, and computer programs, configured to perform theactions of the methods, encoded on computer storage devices.

These and other embodiments can each optionally include one or more ofthe following features. The method may further include accessingperformance logs for the plurality of historical document editingsessions of the plurality of users, each performance log includinghistorical performance measures for the historical document editingsession; and training the suggestion model based on the historicalperformance measures and historical suggestions included in theperformance logs.

Training the suggestion model may include: determining, based on thehistorical performance measures and historical suggestions included inthe performance logs, a relation between a user typing speed and a rateof historical suggestion acceptance; and training the suggestion modelbased on the relation.

Training the suggestion model may include: determining, based on thehistorical performance measures and historical suggestions included inthe performance logs, a relation between a user device latency and arate of historical suggestion acceptance; and training the suggestionmodel based on the relation.

Training the suggestion model may include: determining, based on thehistorical performance measures and historical suggestions included inthe performance logs, a relation between a suggestion confidence scoreand a rate of historical suggestion acceptance; and training thesuggestion model based on the relation.

The one or more performance measures may include one or more of: a usertyping speed; a user device latency; or a rate of suggestion acceptancethat specifies a rate of acceptance for previous suggestions that wereprovided during the document editing session for the prior text that wasincluded in the document prior to the textual input.

Each textual suggestion may have a suggestion confidence scoreindicating a likelihood that the textual suggestion will be included inthe document; the output of the suggestion model may be a confidencescore threshold; and throttling textual suggestions may includeproviding a textual suggestion to the user device only in response tothe textual suggestion having a suggestion confidence score that meetsthe confidence score threshold.

The one or more performance measures may include a rate of acceptancefor previous suggestions that were provided during the document editingsession for the prior text that was included in the document prior tothe textual input; and the confidence score threshold provided by thesuggestion model may depend on the rate of acceptance for previoussuggestions.

The one or more performance measures may include a user device latencythat specifies a communications delay between the user device and thedocument editing application; the output of the suggestion model may bea latency threshold; and throttling textual suggestions may includeproviding textual suggestions to the user device only in response to theuser device latency meeting the latency threshold.

The one or more performance measures may include a user typing speedthat specifies a speed at which the user device provides textual inputto the document editing application; the output of the suggestion modelmay be a typing speed threshold; and throttling textual suggestions mayinclude providing textual suggestion to the user device only in responseto the user typing speed meeting the typing speed threshold.

The document editing application may be one of a plurality of documentediting applications, and the suggestion model was trained usinghistorical performance measures identified in performance logs for thedocument editing application.

The document editing session from which session data is obtained may bethe current editing session.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Providing suggestions for inclusion in a documentmay reduce the need for users to manually draft portions of a document.A user may forget, or be unaware of, various facts or other informationthat the user wishes to include in a document, and a suggestion systemmay be able to assist the user by providing them with information theuser needs, without requiring explicit user requests for assistance.Using suggestion models to predict if and when suggestions should beprovided to a user device may increase the likelihood of helpfulsuggestions being provided to users. Users who respond positively tosuggestions and/or have devices and drafting abilities conducive toreceiving suggestions may receive more suggestions, while users whorespond negatively to suggestions and/or have devices and draftingabilities less conducive to receiving suggestions may receive less, bothof which may improve users' suggestion experience. In addition, theworkload of a suggestion system may be more appropriately matched to theneeds of users, leading to potential efficiency gains for the suggestionsystem.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example environment in which suggestionsare provided for a document.

FIG. 2 is an illustration of an example process for providingsuggestions within a document.

FIG. 3 is a flow diagram of an example process in which suggestions areprovided for a document.

FIG. 4 is a block diagram of an example data processing apparatus.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

A suggestion system provides suggestions, e.g., auto-completions, forusers editing documents. A document is a computer file containing text,such as a word processing document, an e-mail message, a blog post, anSMS or similar text message, or a web page, as well as text entry fieldsin applications and the like. Users may edit documents using a documentediting application, including, for example, a word processorapplication, an e-mail client application, a web-based bloggingapplication, etc. A suggestion system may use information from varioussources to assist a user in drafting and/or editing a document byproviding suggestions. Suggestions may range in size from suggestedcharacters, words, phrases, sentences, paragraphs, formulas,abbreviations, symbols, or more. As used herein, a “word” or “words” mayencompass any of the foregoing, e.g., a suggested “word” may be one ormore characters, words, phrases, sentences, paragraphs, formulas,abbreviations, symbols, etc. Whether suggestions are provided or not,how they are provided, and the content of the suggestions depend onvarious types of information related to, for example, the user editingthe document, existing text included in the document, current text beinginserted by the user, user data related to the user editing thedocument, information regarding other users and/or documents of otherusers, and/or other information.

In an example implementation, a document editing application uses asuggestion system to provide textual suggestions, e.g., autocompletesuggestions, to a user device providing input to the editingapplication. The textual suggestions may depend upon user data for theuser device, and may include both custom and general suggestions, e.g.,a mix of custom suggestions based on historical activity and/orpersonalized user data associated with the user device and generalsuggestions based on either or both of device-independent informationand information associated with multiple user devices. The textualsuggestions may be throttled based on various performance metricsidentified for the current document editing session. Throttlingsuggestions may include, for example, reducing or increasing thelikelihood that a textual suggestion will be provided for the currentediting session or preventing textual suggestions from being provided atall.

In operation, the document editing application, such as a wordprocessing application or e-mail drafting application, obtains, during acurrent editing session, e.g., while the word processing document ore-mail is open for editing, textual input from a user device. Thetextual input may be, for example, characters, words, and phrases thatare inserted into a document or e-mail. To determine whether and/or howoften textual suggestions will be provided to a user device, thedocument editing application may identify session specific performancemeasures for the document during the current editing session.Performance measures may include, for example, a user typing speed, auser device connection speed, and/or an acceptance rate for previoussuggestions, and each performance measure may be identified from priortext that was included in the document before the textual input. A usertyping speed, for example, may indicate the number of words per minutebeing provided by the user device as document input, while the userdevice connection speed may indicate a latency for communicationsbetween the user device and the data processing apparatus running thedocument editing application.

The performance measures may be provided to a suggestion model that istrained to predict whether or not a suggestion should be provided. Themodel may be trained based on historical performance measures obtainedfrom logs of previous user sessions, of the current user and/or otherusers. For example, an increase in typing speed may reduce thelikelihood or frequency of suggestions if historical data indicates thatsuggestions are less likely to be used as typing speed increases. Asanother example, if a connection speed is slow, the likelihood orfrequency of suggestions may be decreased as historical data mayindicate that slower connection speeds lead to less suggestions beingselected by users. Based on the output of the suggestion model, textualsuggestions may be throttled during the current editing session. Forexample, if the performance measures for the current editing sessionindicate that suggestions are rarely selected, the document editingapplication may only provide suggestions that have a high confidence ofbeing selected.

In situations in which the systems discussed here collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures collect user information (e.g., information about a user'ssocial network, social actions or activities, profession, a user'spreferences, or a user's current location), or to control whether and/orhow to receive content from the content item management system that maybe more relevant to the user. In addition, certain data may be treatedin one or more ways before it is stored or used, so that personallyidentifiable information is removed. For example, a user's identity maybe treated so that no personally identifiable information can bedetermined for the user, or a user's geographic location may begeneralized where location information is obtained (such as to a city,ZIP code, or state level), so that a particular location of a usercannot be determined. Thus, the user may have control over howinformation is collected about the user and used by a content itemmanagement system.

These features and additional features are described in more detailbelow.

FIG. 1 is a block diagram of an example environment 100 in whichsuggestions are provided for a document. A computer network 102, such asa local area network (LAN), wide area network (WAN), the Internet, or acombination thereof, connects user devices 104 to a document system 108.The online environment 100 may include any number of user devices 104.In some implementations, connections between user devices 104 and thedocument system 108 may be local, e.g., the document system 108 may bepart of or directly connected to a user device rather than connectedacross the network 102.

A user device 104 is an electronic device capable of requesting andreceiving resources, such as documents, over the network 102. Exampleuser devices 104 include personal computers, mobile communicationdevices, and other devices that can send and receive data over thenetwork 102. A user device 104 typically includes a user application,such as a web browser, to facilitate the sending and receiving of dataover the network 102. The web browser can enable a user to display andinteract with text, images, videos, music, web applications, and otherinformation typically located on a web page at a website.

A document system 108 communicates with one or more user devices 104 toprovide the user devices 104 with access to documents, e.g., byproviding a document editing application interface. For example, thedocument system 108 could be an e-mail server that provides an e-mailinterface through which user devices 104 read and write e-mails, or acloud word processing server that provides an interface through whichuser devices 104 create, modify, and share word processing documents,presentations, and spreadsheets.

A suggestion system 110 provides suggestions for inclusion in adocument. For example, the suggestion system 110 may receive textualinput from a user device, and the suggestion system 110 can use thetextual input to determine whether to provide a suggestion and, if so,identify suggested text to provide to the user device. The suggestionsystem 110 may receive textual input from the document system 108 or, insome implementations, directly from a user device.

Document data 112 is used to store data used by the document system 112and may include, for example, document files, user data, and performancemeasures. The suggestion data 114 is used to store data used by thesuggestion system 110 and may include, for example, an index ofsuggestions, suggestion model training data, and performance measuresfor suggestions. Other information may also be stored in the documentdata 112 and/or the suggestion data 114. While the storage devices aredepicted separately in the example environment 100, in someimplementations some or all of the document data 112 and suggestion data114 may be combined or stored separately in other data storage devices.

Similarly, while the document system 108 and suggestion system 110 aredepicted separately from one another, in some implementations they maybe part of the same system. For example, the suggestion system 110 couldbe a component of the document system 108. In some implementations, thedocument system 108 or a portion thereof, such as a document editingapplication, may be included on a user device. For example, a documentediting application running locally on a user device may communicatewith a document system 108 and/or suggestion system 110 through thenetwork 102.

FIG. 2 is an illustration of an example process 200 for providingsuggestions within a document. The document system 108 receives textualinput 202 from a user device 204. For example, the document system 108may be an e-mail server that provides a document editing applicationthat the user device 204 uses to draft an e-mail, and the textual input202 may be text that the user device provides for inclusion in the bodyof the e-mail, e.g., textual input for the body of an e-mail may be, “Iam going to join Jon and his wi.” In some implementations, the documentfor which the textual input 202 is provided includes prior text that waspreviously included in the document, e.g., entered earlier in thatdrafting session by the user device 204, included in the e-mailautomatically as one or more prior e-mails in an e-mail thread, or savedin a draft e-mail created during a previous editing session.

The document system 108 uses session data 206 for the current editingsession of the user device 204 to identify performance measures 208. Thesession data includes data related to the textual input and also forprior text in the document, e.g., obtained from document data 112. Insome implementations, the session data 206 includes a log for thecurrent editing session of the user device 204. The logs may include,for example, entries for each time the document was edited, including:the type of edit, the latency between the user device 204 and thedocument system 108, and a timestamp for the edit. The logs included inthe session data 206 may also specify when previous suggestions wereprovided to the user device 204, and whether or not the suggestions wereaccepted. By way of example, session data 206 for an e-mail beingdrafted by the user device 204 may include a log with entries for eachcharacter added or removed to the e-mail, including entries for thetextual input, e.g., “I am going to join Jon and his wi.”

Using the session data 206, the document system identifies theperformance measures 208 for the current editing session. In the exampleprocess 200, the example performance measures 208 are: latency of 102milliseconds (ms), a typing speed of 24 words per minute (wpm), and anacceptance rate of 0.75, e.g., a number of suggestions accepted persuggestions provided, on a scale of 0 to 1.

The performance measures 208 are provided to the suggestion system 110to determine whether suggestions should be provided, and if so, underwhat circumstances. The suggestion system 110 determines whethersuggestions for the current editing session should be throttled usingthe performance measures 208 and a suggestion model that was trainedusing historical performance measures identified in performance logs 210of historical document editing sessions of multiple users.

Each of the performance logs 210 includes historical performancemeasures for a historical document editing session, such as latency,typing speed, suggestion confidence scores for suggestions presented,and a suggestion acceptance rate. In some implementations, thesuggestion model is trained by identifying relations between latency andacceptance rate, typing speed and acceptance rate, and suggestionconfidence score to suggestion acceptance rate. Other model relationsmay also be used to train the model, such as a relation between a rateof prior of suggestion acceptance and a rate of later suggestionacceptance, e.g., how does declining or accepting suggestions early in adocument affect the rate at which later-presented suggestions areaccepted.

In some implementations, training the suggestion model includesdetermining a relation between a user typing speed and a rate ofsuggestion acceptance. For example, users who type slowly may be morelikely to see, read, and accept suggestions than users who type quickly;users who type quickly may not see the suggestion or may be typing tooquickly to read and accept a presented suggestion.

In some implementations, training the suggestion model includesdetermining a relation between user device latency and a rate ofsuggestion acceptance. User device latency measures the delay incommunications between the user device 204 and one or more othersystems, such as the document system 108 and/or the suggestion system.For example, a user device with low latency, e.g., 80 ms, will bepresented with suggestions faster than a user device with high latency,e.g., 1,000 ms. Latency may affect how quickly suggestions are presentedat a user device, which may in turn affect how often they are presentedand/or accepted. A low latency may also lead to performance problems,which may reduce the desirability of providing suggestions. For example,in a cloud computing environment, user devices with high latency willexperience a delay between actions performed on the user device andresponses from the document system, and providing suggestions mayfurther exacerbate the delay.

In some implementations, the suggestion model is trained by determininga relation between a suggestion confidence score and a rate ofhistorical suggestion acceptance. Suggestions provided by the suggestionsystem 110 may have a corresponding confidence score that indicates ameasure of confidence that the suggestion will be accepted by a user.Suggestions with high suggestion confidence scores may, for example, bemore likely to be accepted than suggestions with low confidence scores.

In some implementations, the suggestion model is trained by determininga relation between a rate of prior suggestion acceptance and a rate oflater suggestion acceptance. For example, the rate at which a useraccepts suggestions in the first portion of a document may correlatewith the rate at which the user accepts suggestions presented in laterportions of the document, e.g., if a user is accepting every suggestionprovided, the user may be likely to accept suggestions presented laterin the editing session for the same document as well, while a userdeclining most suggestions provided early in a document editingsuggestion may be likely to ignore or decline suggestions presentedlater.

Combinations of one or more of the foregoing historical performancemeasures may be used to train the suggestion model used to determinewhether and how suggestions should be throttled. Other performancemeasures and relations may also be used, alone or in combination withthose described above, to train the suggestion model.

The model output 212 produced by the suggestion model indicates themanner in which textual suggestions are throttled. As noted above,throttling suggestions may include, for example, reducing or increasingthe likelihood that a textual suggestion will be provided for thecurrent editing session or preventing textual suggestions from beingprovided at all. Throttling may also include adjusting confidence scoresfor suggestions or, in situations where confidence score thresholds areused to determine whether a suggestion should be provided, raising orlowering a confidence score threshold.

For example, in implementations where the session specific performancemeasures 208 include user device 204 latency, the output of thesuggestion model may be a latency threshold that must be met before anysuggestions are provided. For example, the suggestion system 110 maydetermine, based on the performance logs 210, that user devices withlatency greater than 1,000 ms should not receive suggestions. If theperformance measures 208 indicated latency greater than 1,000 ms, thesuggestions system 110 or suggestion model may provide model output 212to the document system indicating that the user device 204 should notrequest or receive textual suggestions until the latency for the usersession drops below 1,000 ms.

As noted above, in some implementations, each suggestion 214 identifiedby the suggestion system 110 based on the textual input 202 includes aconfidence score. In these implementations, the model output 212produced by the suggestion model may be a confidence score threshold,and suggestions 214 are throttled by only providing suggestions 214having a confidence score greater than the threshold. For example, modeloutput may indicate a relatively high confidence score threshold foruser devices with high latency, and a relatively low confidence scorethreshold for user devices with low latency.

In some implementations, as in the example process 200, the sessionspecific performance measures 208 may include a rate of suggestionacceptance for suggestions that were previously provided to the userdevice 204 during the current document editing session, and a confidencescore threshold provided by the suggestion model may depend on the rateof acceptance for previous suggestions. For example, the model output212 may specify a low confidence score threshold if the user isaccepting most suggestions provided during the current editing session,which may lead to an increase in the number of suggestions provided.

In some implementations, as in the example process 200, the sessionspecific performance measures 208 may include a user typing speed, andthe suggestion model may provide a typing speed threshold as modeloutput 212, e.g., instead of or in addition to a confidence scorethreshold. As with the latency threshold, a typing speed threshold mayprevent suggestions from being provided as long as the user's typingspeed meets the typing speed threshold. In some implementations, thenumber of suggestions provided may be reduced or increased as the user'styping speed changes. For example, users who type quickly may see fewersuggestions than users who type slowly.

Other methods or combinations of methods, in addition to those describedabove, may be used to throttle textual suggestions. The suggestion modelmay receive multiple performance measures 208 as input to a singlesuggestion model, or may use separately trained models with individualperformance measures being input to each, with the results beingcombined. For example, the suggestion model that receives latency inputmay produce model output 212 indicating that the frequency or likelihoodof suggestions 214 should be increased for a latency of 102 ms, whilethe suggestion model that receives typing speed input may produce modeloutput 212 indicating that the frequency or likelihood of suggestions214 should be decreased for a typing speed of 84 wpm or greater.

In some implementations, the model output 212 used to make adjustmentsaffecting the frequency, or likelihood, or suggestions 214 beingprovided may be used for an entire user editing session, or may beupdated periodically and/or on demand, e.g., with each new request for asuggestion. In addition, the performance measures 208 may be measuredand re-measured periodically, or on demand. For example, the performancemeasures may be updated every second, every 10 seconds, after everysuggestion, or after receiving N keystrokes as input from the userdevice, where N is a positive integer. In addition, the suggestion modelmay, in some implementations, be periodically retrained using differentperformance logs 210.

In some implementations, a user may be provided with an opportunity tocontrol the throttling of suggestions. For example, a user may havepreferences indicating how frequently the user wishes to seesuggestions. Preferences may be context specific, e.g., resulting indifferent preferences for different devices, applications, locations,etc. Preferences can affect, for example, the sensitivity ofsuggestions, e.g., by adjusting a threshold confidence score fordelivering suggestions, and the frequency of suggestions, e.g., byturning suggestions off for a particular user device.

In some implementations, other indicators may be used to determinewhether or not suggestions 214 should be provided to the user device204. For example, a determination may be made based on the content ofthe textual input, e.g., by determining that the textual input includesa misspelling, determining that the textual input includes a referenceto an entity known to the suggestion system 110, or determining that thetextual input includes a special character or combination of charactersdesigned to trigger a suggestion.

The suggestion system 110 may use the textual input 202 and one or moresuggestion models, such as a general suggestion model, to identify theactual suggestion(s) 214 to be provided to the user device 204. Forexample, a general suggestion model is a model that has been trained,based on text included in historical documents, to identify textualsuggestions for completing a word, phrase, sentence, paragraph, etc.based on textual input 202. For example, given the textual input 202 of“I am going to join Jon and his wi,” the general suggestion model mayprovide one or more general suggestions 214, such as “wife” or“wildlife.”

In some implementations, the general suggestion model also provides aconfidence score for each general suggestion. For example, a confidencescore for the suggestion “wife” may be based on how often other wordsbeginning with “wi” turned out to be “wife.” If “wife” is more common inhistorical text than “wildlife,” then the confidence score of “wife” maybe higher than the confidence score for “wildlife.” Many other methods,or combinations of methods, may be used to determine confidence scoresfor completions. In some implementations, partial or complete phrasematches, rather than the most recent word or characters, may be used toidentify general suggestions and measure their confidence, e.g., havingthe pronoun “his” before “wi” may increase the confidence score for“wife” relative to “wildlife” if historical instances of “his wife” aremore common than “his wildlife.”

Many different methods or combination of methods may be used todetermine which suggestions identified by the suggestion system 110should be provided to the user device 202. In implementations where thesuggestions have confidence scores, the suggestion with the highestconfidence score may be selected for presentation. In someimplementations, the suggestions 214 may be ranked according to theirconfidence scores and the top N are selected for presentation, where Nis a positive integer. As noted above, one or more thresholds may beused in some implementations, e.g., suggestions may only be selected iftheir respective confidence scores meet a confidence score threshold.

As depicted in the example process 200, the selected suggestion(s) 214may be provided to the user device 204. For example, the document system108 may cause a document editing application to present selectablesuggestion(s) 214 to the user, as in a drop-down list, or it may replaceexisting characters of the document with a suggestion and a notificationregarding the replacement.

While various components, such as the document system 108 and suggestionsystem 110, are depicted separately in the illustration of the exampleprocess 200, the components may be included in a single system, as shownby the dotted line encompassing the components, or a differentcombination of systems than the depicted combination. In addition, thestorage devices depicted may be combined, and may also be stored, inwhole or in part, separately from the system that provides suggestions.

FIG. 3 is a flow diagram of an example process 300 in which suggestionsare provided for a document. The process 300 may be performed by asuggestion system, such as the system described above with reference toFIG. 2. Process steps 302 and 304 may be performed prior to query time,such as during model training.

In some implementations, performance logs are accessed for historicaldocument editing sessions of users (302). Each performance log includesperformance measures for a historical document editing session. Forexample, a performance log for a word processing document may includeinformation indicating how fast the user drafting the document wastyping at any given time, how many suggestions were provided andaccepted, and a latency of the connection at any given time.

In some implementations, a suggestion model is trained based on theperformance measures and historical suggestions included in theperformance logs (304). Training the suggestion model may includedetermining, based on the performance measures and historicalsuggestions included in the performance logs, a relation between a usertyping speed and a rate of historical suggestion acceptance. Thesuggestion model may then be trained based on the relation. For example,the suggestion model may be trained to decrease the likelihood ofsuggestions being provided as typing speed increases.

In some implementations, training the suggestion model includesdetermining, based on the performance measures and historicalsuggestions included in the performance logs, a relation between a userdevice latency and a rate of historical suggestion acceptance. Thesuggestion model may then be trained based on this relation. Forexample, the suggestion model may be trained to decrease the likelihoodor frequency of providing suggestions as latency increases.

In some implementations, training the suggestion model includesdetermining, based on the performance measures and historicalsuggestions included in the performance logs, a relation between asuggestion confidence score and a rate of historical suggestionacceptance. The suggestion model may then be trained based on thisrelation. For example, the suggestion model may be trained to determinea threshold confidence score for providing a suggestion, and suggestionsmust have a confidence score that meets the threshold in order to beprovided to a user device.

In some implementations, training the suggestion model includesdetermining, based on the performance measures and historicalsuggestions included in the performance logs, a relation between a rateof prior suggestion acceptance and a rate of later suggestionacceptance. The suggestion model may then be trained based on thisrelation. For example, the model may be trained to determine a measureof confidence that a suggestion will be accepted based on the rate atwhich previous suggestions for the same document were accepted.

Textual input provided to a document editing application by a userdevice, during a current editing session, is obtained (306). The textualinput is provided to the document editing application for inclusion in adocument. For example, the document editing application may be a wordprocessing document editing application, and the textual input may betext recently entered by a user device, e.g., the previous N words,where N is a positive integer, or the words entered since the lastpunctuation mark.

One or more performance measures are identified for the document duringthe current editing session (308). Each performance measure is based onsession data obtained from the user device, the session data being forthe textual input and prior text that was included in the document priorto the textual input. For example, in some implementations theperformance measures may include one or more of a user typing speed, auser device latency, and/or a rate of suggestion acceptance thatspecifies a rate of acceptance for previous suggestions that wereprovided during the current editing session for the prior text that wasincluded in the document prior to the textual input. The documentsession from which session data is obtained may, in someimplementations, be the current editing session. For example, the usertyping speed, user device latency, and/or rate of suggestions acceptancemay each be for the current user session.

The one or more performance measures are provided as input to asuggestion model (310). The suggestion model has been trained usinghistorical performance measures identified in performance logs forhistorical document editing sessions of users. For example, performancelogs may be maintained for a large collection of word processingdocuments drafted by various users. As described above, the informationincluded in the performance logs, such as user typing speed, latency,document changes, and suggestions provided and accepted, may have beenused to train the suggestion model.

Textual suggestions are throttled during the current editing sessionbased on the output of the suggestion model (312). Each textualsuggestion includes one or more words to be suggested for inclusion inthe document. In some implementations, each textual suggestion has asuggestion confidence score indicating a likelihood that the textualsuggestion will be included in the document, the output of thesuggestion model is a confidence score threshold, and throttling thetextual suggestions includes providing a textual suggestion to a userdevice only in response to the textual suggestion having a suggestionconfidence score that meets the confidence score threshold. For example,a user may begin typing a sentence as follows: “The main character ofthe New Awesome Movie is played by.” A suggestion system may identify anindividual, e.g., “John Doe,” as a suggestion, with a confidence scoreof 0.90 (on a scale of 0 to 1). If a confidence score threshold is 0.80,the suggestion meets the confidence score threshold and is eligible tobe provided to the user device, while a confidence score threshold of0.95 would make the suggestion ineligible for provision to the userdevice. The confidence score may, in some implementations, depend atleast in part on characters in a prefix typed by a user. For example,the confidence score for “John Doe” in the previous example may behigher if the user input includes additional characters, e.g., “The maincharacter of the New Awesome Movie is played by Joh” may result in ahigher confidence score for “John Doe” than “The main character of theNew Awesome Movie is played by.”

In some implementations, the one or more performance measures include arate of acceptance for previous suggestions that were provided duringthe current editing session for the prior text that was included in thedocument prior to the textual input. The confidence score thresholdprovided by the suggestion model may depend on the rate of acceptancefor previous suggestions. For example, if the user providing the textualinput, “The main character of the New Awesome Movie is played by,” waspreviously presented with 10 suggestions during the same documentediting session for the same document, the number of suggestionsaccepted by the user may affect the confidence score threshold. Forexample, if the user accepted all 10 suggestions, this may indicate thatthe user finds the suggestions useful, and the suggestion threshold maybe decreased to allow for more suggestions to be eligible to be providedto the user device. On the other hand, if the user declined all 10previous suggestions, this may indicate that the user does not find thesuggestions useful, and the suggestion threshold may be increased toensure that only suggestions with high confidence scores are provided tothe user device.

In some implementations, the one or more performance measures include auser device latency that specifies a communications delay between theuser device and the document editing application. The output of thesuggestion model may be a latency threshold, and throttling textualsuggestions may include providing textual suggestions to the user deviceonly in response to the user device latency meeting the latencythreshold. For example, if a user device has a high latency, identifyingand providing suggestions may be delayed, and the user may find thesedelays unhelpful. In these situations, a latency threshold may be usedto prevent suggestions from being provided unless and until the userdevice latency is less than the threshold latency.

In some implementations, the one or more performance measures includes auser typing speed that specifies a speed at which the user deviceprovides textual input to the document editing application. The outputof the suggestion model may be a typing speed threshold, and throttlingthe textual suggestions may include providing textual suggestions to theuser device only in response to the user typing speed meeting the typingspeed threshold. For example, users who type quickly may be less likelyto read and accept suggestions, as they may type faster than suggestionscan be presented and read. In these situations, a typing speed thresholdmay be used to prevent suggestions from being provided to the user. Ifand when the user typing speed decreases, suggestions may then beprovided to the user device. In addition, the frequency of suggestionsmay be increased as typing speed decreases, which may improveproductivity of slow typing users.

In some implementations, the document editing application is one ofmultiple document editing applications, and the suggestion model wastrained using historical performance measures identified in performancelogs for that particular document editing application. In addition,different suggestion models may be used for different document editingapplications. For example, an e-mail authoring application may have asuggestion model trained using e-mail documents and used for e-maildocuments, and that e-mail suggestion model may be different fromsuggestion model for a word processing document drafting application. Insome implementations, other context-dependent models and/or model inputsmay be used to determine whether suggestions will be provided forvarious contexts. For example, the hardware or software used forproviding input may affect the provision of suggestions, e.g., a modelused for textual input provided using swipe-typing software may bedifferent from a model used for textual input provided using a physicalkeyboard.

FIG. 4 is a block diagram of an example data processing apparatus 400.The system 400 includes a processor 410, a memory 420, a storage device430, and an input/output device 440. Each of the components 410, 420,430, and 440 can, for example, be interconnected using a system bus 450.The processor 410 is capable of processing instructions for executionwithin the system 400. In one implementation, the processor 410 is asingle-threaded processor. In another implementation, the processor 410is a multi-threaded processor. The processor 410 is capable ofprocessing instructions stored in the memory 420 or on the storagedevice 430.

The memory 420 stores information within the system 400. In oneimplementation, the memory 420 is a computer-readable medium. In oneimplementation, the memory 420 is a volatile memory unit. In anotherimplementation, the memory 420 is a non-volatile memory unit.

The storage device 430 is capable of providing mass storage for thesystem 400. In one implementation, the storage device 430 is acomputer-readable medium. In various different implementations, thestorage device 430 can, for example, include a hard disk device, anoptical disk device, or some other large capacity storage device.

The input/output device 440 provides input/output operations for thesystem 400. In one implementation, the input/output device 440 caninclude one or more network interface devices, e.g., an Ethernet card, aserial communication device, e.g., an RS-232 port, and/or a wirelessinterface device, e.g., an 802.11 card. In another implementation, theinput/output device can include driver devices configured to receiveinput data and send output data to other input/output devices, e.g.,keyboard, printer and display devices 460. Other implementations,however, can also be used, such as mobile computing devices, mobilecommunication devices, set-top box television client devices, etc.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus.

A computer storage medium can be, or be included in, a computer-readablestorage device, a computer-readable storage substrate, a random orserial access memory array or device, or a combination of one or more ofthem. Moreover, while a computer storage medium is not a propagatedsignal, a computer storage medium can be a source or destination ofcomputer program instructions encoded in an artificially-generatedpropagated signal. The computer storage medium can also be, or beincluded in, one or more separate physical components or media (e.g.,multiple CDs, disks, or other storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing. The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., a FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's user device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., auser computer having a graphical user interface or a Web browser throughwhich a user can interact with an implementation of the subject matterdescribed in this specification, or any combination of one or more suchback-end, middleware, or front-end components. The components of thesystem can be interconnected by any form or medium of digital datacommunication, e.g., a communication network. Examples of communicationnetworks include a local area network (“LAN”) and a wide area network(“WAN”), an inter-network (e.g., the Internet), and peer-to-peernetworks (e.g., ad hoc peer-to-peer networks).

The computing system can include users and servers. A user and serverare generally remote from each other and typically interact through acommunication network. The relationship of user and server arises byvirtue of computer programs running on the respective computers andhaving a user-server relationship to each other. In some embodiments, aserver transmits data (e.g., an HTML page) to a user device (e.g., forpurposes of displaying data to and receiving user input from a userinteracting with the user device). Data generated at the user device(e.g., a result of the user interaction) can be received from the userdevice at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A method implemented by a data processingapparatus, the method comprising: obtaining, during a current editingsession of a document, textual input provided to a document editingapplication for inclusion in the document by a user associated with auser device; identifying, during the current editing session, one ormore performance measures associated with the current editing session ofthe document, each performance measure being based on session dataobtained from the user device and at least one of the performancemeasures being typing speed of the user during the current editingsession, the session data including data related to (i) the obtainedtextual input and (ii) prior textual data that was included in thedocument prior to the obtained textual input; receiving textualsuggestions based on the textual input, each textual suggestion receivedfrom a suggestion model and having a respective confidence score, eachtextual suggestion comprising one or more words to be suggested forinclusion in the document; providing the one or more performancemeasures as input to the suggestion model that has been trained usinghistorical performance measures identified in performance logs for aplurality of historical document editing sessions of a plurality ofusers and prior to the current editing session; generating, by thesuggestion model, a threshold value based on the one or more performancemeasures that were provided as an input to the suggestion model; andproviding, during the current editing session, textual suggestions thathave respective confidence scores greater than the threshold value. 2.The method of claim 1, the method further comprising: training thesuggestion model, wherein training the suggestion model comprisestraining the suggestion model based on historical suggestions includedin the performance logs.
 3. The method of claim 2, wherein the methodfurther comprises: determining, based on the historical performancemeasures and the historical suggestions included in the performancelogs, a relation between the user's typing speed and a rate ofhistorical suggestion acceptance; and wherein training the suggestionmodel further comprises: training the suggestion model based on thedetermined relation.
 4. The method of claim 2, wherein the methodfurther comprises: determining, based on the historical performancemeasures and the historical suggestions included in the performancelogs, a relation between a suggestion confidence score and a rate ofhistorical suggestion acceptance; and wherein training the suggestionmodel further comprises: training the suggestion model based on thedetermined relation.
 5. The method of claim 1, wherein the documentediting application is an e-mail application.
 6. The method of claim 1,wherein the one or more performance measures further comprise a rate ofsuggestion acceptance that specifies a rate of acceptance for previoussuggestions that were provided during the plurality of historicaldocument editing sessions.
 7. The method of claim 1, wherein thedocument editing application is one of a plurality of document editingapplications, and the suggestion model was trained using respectivehistorical performance measures identified in respective performancelogs for the document editing application.
 8. A system comprising: oneor more data processing apparatus implemented at least partially byhardware; and a data storage device storing instructions that, whenexecuted by the one or more data processing apparatus, cause the one ormore data processing apparatus to perform operations comprising:obtaining, during a current editing session of a document, textual inputprovided to a document editing application for inclusion in the documentby a user associated with a user device; identifying, during the currentediting session, one or more performance measures associated with thecurrent editing session of the document, each performance measure beingbased on session data obtained from the user device and at least one ofthe performance measures being typing speed of the user during thecurrent editing session, the session data including data related to (i)the obtained textual input and (ii) prior textual data that was includedin the document prior to the obtained textual input; receiving textualsuggestions based on the textual input, each textual suggestion receivedfrom a suggestion model and having a respective confidence score, eachtextual suggestion comprising one or more words to be suggested forinclusion in the document; providing the one or more performancemeasures as input to the suggestion model that has been trained usinghistorical performance measures identified in performance logs for aplurality of historical document editing sessions of a plurality ofusers and prior to the current editing session; generating, by thesuggestion model, a threshold value based on the one or more performancemeasures that were provided as an input to the suggestion model; andproviding, during the current editing session, textual suggestions thathave respective confidence scores greater than the threshold value. 9.The system of claim 8, the method further comprising: training thesuggestion model, wherein training the suggestion model comprisestraining the suggestion model based on historical suggestions includedin the performance logs.
 10. The system of claim 9, wherein the methodfurther comprises: determining, based on the historical performancemeasures and the historical suggestions included in the performancelogs, a relation between the user's typing speed and a rate ofhistorical suggestion acceptance; and wherein training the suggestionmodel further comprises: training the suggestion model based on thedetermined relation.
 11. The system of claim 9, wherein the methodfurther comprises: determining, based on the historical performancemeasures and the historical suggestions included in the performancelogs, a relation between a suggestion confidence score and a rate ofhistorical suggestion acceptance; and wherein training the suggestionmodel further comprises: training the suggestion model based on thedetermined relation.
 12. The system of claim 8, wherein the one or moreperformance measures further comprise a rate of suggestion acceptancethat specifies a rate of acceptance for previous suggestions that wereprovided during the plurality of historical document editing sessions.13. The system of claim 8, wherein the document editing application isone of a plurality of document editing applications, and the suggestionmodel was trained using respective historical performance measuresidentified in respective performance logs for the document editingapplication.
 14. A non-transitory computer-readable medium comprisinginstructions that, when executed by one or more data processingapparatus, cause the one or more data processing apparatus to performoperations comprising: obtaining, during a current editing session of adocument, textual input provided to a document editing applicationduring a current editing session for inclusion in the document by a userassociated with a user device; identifying, during the current editingsession, one or more performance measures associated with the currentediting session of the document, each performance measure being based onsession data obtained from the user device and at least one of theperformance measures being typing speed of the user during the currentediting session, the session data including data related to (i) theobtained textual input and (ii) prior textual data that was included inthe document prior to the obtained textual input; receiving textualsuggestions based on the textual input, each textual suggestion receivedfrom a suggestion model and having a respective confidence score, eachtextual suggestion comprising one or more words to be suggested forinclusion in the document; providing the one or more performancemeasures as input to the suggestion model that has been trained usinghistorical performance measures identified in performance logs for aplurality of historical document editing sessions of a plurality ofusers and prior to the current editing session; generating, by thesuggestion model, a threshold value based on the one or more performancemeasures that were provided as an input to the suggestion model; andproviding, during the current editing session, textual suggestions thathave respective confidence scores greater than the threshold value. 15.The computer-readable medium of claim 14, the method further comprising:training the suggestion model, wherein training the suggestion modelcomprises training the suggestion model based on historical suggestionsincluded in the performance logs.
 16. The computer-readable medium ofclaim 15, wherein the method further comprises: determining, based onthe historical performance measures and the historical suggestionsincluded in the performance logs, a relation between the user's typingspeed and a rate of historical suggestion acceptance; and whereintraining the suggestion model further comprises: training the suggestionmodel based on the determined relation.
 17. The computer-readable mediumof claim 15, wherein the method further comprises: determining, based onthe historical performance measures and the historical suggestionsincluded in the performance logs, a relation between a suggestionconfidence score and a rate of historical suggestion acceptance; andwherein training the suggestion model further comprises: training thesuggestion model based on the determined relation.
 18. Thecomputer-readable medium of claim 14, wherein the one or moreperformance measures further comprise a rate of suggestion acceptancethat specifies a rate of acceptance for previous suggestions that wereprovided during the plurality of historical document editing sessions.19. The computer-readable medium of claim 14, wherein the documentediting application is one of a plurality of document editingapplications, and the suggestion model was trained using respectivehistorical performance measures identified in respective performancelogs for the document editing application.